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INTRODUCTORY COMMENTS 



In this paper, Mlliam J* Webster provides educators with a' rjiview of 
current , th&ory on* instructional materials evaluation^ aj^ong with a brief 
description of how a practical system of materials evaluation b.^s been applied 
in a large urban school district. As a result^ local educational decision- 
4,Tnakers who are interested in improving their kn6v;ledge of the still-infant 
discipline of instructional materials evaluation are given aiL opportunity to 
acquaint themselves t^ith what amounts to a theoretical/praotical description 
of the advanced state of the art of materials evaluation. 

The designation "advanced" is certainly appropriate when one compares 
the approaches described in this paper with what passes for materials 
evaluation in most school systems.. Indeed, if as many as five percent of 
this country's l^yOOO-plus school systeins are using anything approaching 
the systematic, ©npirically-based discrepancy model described by Dr. Wetaster, 
this writer will happily eat every available evaluation report, staples' and 
all* In fact, one of the la:rge cliscrepancies in toierican education today 
is between the enormous effect inKtructional materials have on classroom 
instruction and the little time, effort, and money expended by school 
districts in ariswering these questions i P3hich materials shall we use? How 
shall teachers use them? For which students are specific materiaiS best 
suited? ; - . 

The're is ample evidence of this discrepancy. A recent nationwide study 
by EPIE, the Educational Products information Exchange (National Survey' and ' ^ 
Assessment of Instructional Materials, 1974-75) , summarizes reports ^^riWjome 
13,000 teachers- within 9,000 school buildings indicating that 90^95 p.erc^t . 
of their classroom, time involves the use of instructional materials of qni ^ * 
sort or another. Yet most of these teachers spend little or no time (one to 
three hours per year) selecting the materials they are using* At the school 
system level, there seldom is any systeinatic information being gathered that 
could be used for the purpose of materials evaluation if, indeed, there were 
the commitment to engage in data^based empirical evaluations. 

What usually passes for evaluation in rchools today is some form of a 
"materials selection checklist"^ or a set of "materials selection guidelines" — 
whose validity and reliability are invariably questionable. The result is 
that most school systems are basi.'^g what amounts to major instruntional ( 
decisions on simple descriptive information that has been ^more or less idio- 
syncratically organized from school system to school system* Even in states - 
ithat attempt to evaluate materials there is little recourse to empirical 
evaluation either before or following" statewide adoption of matarlals. Here 
too the reliance on the ubiquitous checklist is heavy. Dr. Webster's paper , ^ 
clearly points to the shortcomings of this reliance* ' 

/ The question implicitly raised by this paper isi Is it realistic to 
assmne, or even to hope, that most-school districts will adopt in the near 
future a materials evaluation system of the sort described by DrJ Webster? 
It would seem that before this can happen there will need to be massive 
consciousness-.raising about the role and the importance of instructional^ 



i 



niaterials on the part of school consumers. The objective of this consciousness- . 
raising would not be to improve materials evaluation/ per se, but to improve, 
the all important three-way "fit" across ntatGrials^, leatners, and teachers. 

Dr. Webster's report in his practical example that "it was discpvered 
\ that certain types of students do better in some programs when taught by 
■ Specific types of teachers" is tantarizing* After a decade of research on the 
"fit" question in insrructional materials evaluation ^ we at EPIE know that we 
have only begun to scratch the surface. . Currently/ EPIE * s research on Q 
materipls-to-learner-to-teacher fit is being eKpanded to include data^gathering 
on instructional materials use and performance across a national network of 
school systems. At present 22 systems are involved^ ranging in size from 
,2,900 to 250, dOO students, with the broadest sort of socioeconomic and ethnic 
spread. -Lhis growing network is a permanent part of EPIE's longitudinal 
National Survey and Assessment of Instructional Materials which completed its 
first biennial qycle in June 1976. Using a model similar to that described 
. by Dr. Webster^ EPIE hopes that this network of cooperating school districts 
will contribute to the much rteeded cbnsciousness-raisrng on this matter that 
must occur if instruction is to be consistfently and rationally improved in 
American schools^ ^ , V ^ 

' ' P. Kenneth Komoski 

, Educational Products Information 
Exchange Institute 
. 475. Riverside Drive 

\ • kew York^ New York 10027 

/ ■ . ■ 
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PURPOSE OF THIS PAPER 



The evaluation of instruGtional materials is an extremely important yet 
of tan overlooked I component of the total instructional process. Often the term 
evaluation is operationally defined as "checklist" by many cirrifjulum specialists 
who. parform a: weak form of input evaluation on instructional .nat jrials, using 
soina variation of a survey form or check-list, in presentina ilcerna^ives to 
inadequate evaluation^ [this paper attempts^ to accomplish three objectives i 



/ 



To synthesiser through a selective review of the ^ 
literat^r^> a brief description of the state of the / 
art of evaluation; ' < / 

- --t "^^ . ■ :■; ^ I 

To present ^ working model demonstrating the functions / 
of various forms of evaluation in assessing the relative 
merits of ins true tional materials i 



.Four Evaluation Framevrorks 



The CIPP Model , 



To provide an annotated bibliography of sources for 
readers seejcing further information on evaluation. 

■ ^ ■ . / 

/ A BRIEF STATE OF THE ART / 

/ 



P^obaJbiy the most comprehensive of existing evaluation 
^1 developed by . Stuff lebeam, et al^ (1971)/ Evaluat^ior 



pdels is the CIPP Mod^l developed by . Stuff lebeam, et ^1^ (1971)/ Evalua^ion_ 
is defined ^~s the process of delineating^ obtaioing, and providing useful^ 
inforffiatio^ for judging decision alternatives^ Me-^iodel idanti foui major 

types of evaluation I Context evaluation to feed planning decisions * input 
evaluation to feed programming decisions , process evaluation to feed tmplamknting 
decisions/ and product evaluation to feed recycling decisions* 

Briefly^ contektf evaluation provl^des a rationale for determining educational.^; 
objectives by defining relevant envirohment , describing desired and actual con-- . 
ditions of the environment, identifying unmet needs # and diagnosing problems thatt 
prevent needs from being met* Input evaluation assesses relevant capabilities 
of responsible agencies, Maentifies strategies for achieving , the objectives- , 
determined through contexff^ evaluation, and suggests designs for implementing 
selected strategies. Once a strategy has been selected, process evaluation 
provides periodic feedback to. help predict or detect faults in procedural design ^ 
or implementation so that interim adjustments may be. made. Finally^ product^ 
evaluation provides interim and final assessment of the effects of ©ducationai 
programs. Th^t is, product evaluation assesses the effects of the- 'strategy . 
selected through in^ut evaluation to meet the need identified by context evaluation 
Such assessment is completed in light of process evaluation data. 

Scriven ^. Scriven (1967) has conceptualized an extremely straightforward 
and widely accepted evaluation frMiework* Not nearly as comprehensive as the' _ 
dIPP Model, it is largely concerned with the proce'ss^product portion of 
Stuff lebeam* s structure* According to Scriven, the major gpal of evaluation is ' 
to credibly j^dge the merits of educational programs* To accomplish the goal,, 
he introduces the concepts of formative and smnmative evaluation, ^ 



\ 



The focus of formative eyaluation is upon program improvement, Thus^ 
formative, evaluation attempt^" to provide feedback to program personnel in order 
tOtUpgrade or improve an educational program whil6 it is in the developmental^ 
stagB* In the Cipp vernacular, interim product'ind process data provide "formative 
evaluation information to program persQnnel=r~ ' ■ ... 

\ - ■ ■ 7 / \ . ' ■ ■ 

JTHa focus of summative evaluation is upon the determination o 2 the ultimate, 
worth o^^a prograin or project. Thik type of evaluatioh should be implemented 
when a program ha.s re^chedl some ^ stability * Oumniative data feed recycling decisions 
as a result of summative evaluation information, a program may be terminated, 
restructur^clf continued, jop. eKpanded. In the CIPP vernacular, final product 
evaluation information, /in:Berpreted in Gonsideration of context^ input/ and process 
data, is usedXto draw siommative conclusions about the merits of an educational ~ 
program and to ^feed recycling decisions. " ^ . . . 

' ' Stake. Stake\^(19^'7) suggests that eyaLuatipn bught to be concerned with 
three classes of c^clitions> antecedents, transactions, and outcomes. | Antecedents 
are those conditions /that exist prior^ to program implementation, i-e^, the 
educational context .X Transactions are interactions between students, teachers, 
and materials. Outqomos are the intendedxprbducts of transactions* 

Three classes of activities are suggested ^y Stake, The first provides 
assistance to program staff by generating a clear statement of the program or 
project rationale/ The se)ipnd activity generates descriptive data# including 
statements about intended arid actual antecedents * transactions ^ and 'Outcomes * 
Thus, aongruence /between pl£mned and observed antecedents / transactions , and 
outcomes can be^c*hecked. \ . " 

The third class of acti\rities generates judgments about the worth of 
educational pTOgrams.^ Stake suggests tiiat such judgments be made by a variety ■ 
of individuals on the basis of both absolute and relative criteria^ In other 
words, programs should be assessed in terms of i 1) The degree to which they 
^attain absolute and sometimes arbitrary goals, and 2) The degree to which they % 
attain those goals relative to other programs' with similar goals or objectives* 

Provus* Provu^ (1971) suggests thatXall projects move through design # in^ 
stallation, processi and product stages* \During each staje the evaluator must 
delineate, in conjunction with project sta^f* a set of standards that can be 
used as a basis for comparison with pragramvperfoirniance. It ,is the evaluator *s 
function to make comparisons with program perlPorm'ancei to identify discrepancies | 
at each stage, and to; report those ^ discrepanca^s to project management who have 
the option of terminating the program; proceedMg to the next stage # or modifying 
the .program* The product of the desigri stage i^ a set of standards used to judge 
the ef^Eeots of ptogram efforts in each of the thr^e succeeding stages* ftt every 
stage the object of the evaluation is to provide ufeful data for decisions about 
program improvement or recycling* \ ' 

General Discussion - x 1' 

^ ■ . ■ . ^- ; ■ 

These four generic evaluation frameworks are supplemented b^, many articles 
dealing with methodology, purposes, instrumer^tationi strategies, and vari^les 
to be considered in evaluation. Among these are the^Dif ferential Evaluation' 

; : ^ ' ■■■■ ■ ■■ ■ 1 . ' ' ■■ ■ ' • 



Model (Tripodi, Fellin, and Epstein^, 1971)., the Decision Oriented Classifica- 
tion jjodfl (Alkin and Wooley, 1959) , \the Apex Evaluation Model (Morgan, 1970), 
the Synergistic Evaluation Model (Hunter and Schooley, 1973), the Ontologicai 
Evaluation Model (Pepar, 1973), and ^hi^Ott MddeLJOtt, Flstcher, and Turner, 
•1966)* ' In addition,, the IPI Formative EYaluation Model jL.lndvall and Cox, 1970) 



were 



and the New Start Evaluation system (Lamrock, Smith, and Warren, 1971) 
developed specifically for the evaluation ^qf individualized curriculum packages 
while the Trade-off and Comparative Cost Approach (Glass, 1972) and the Weighted 
Criteria Approach (Crane and Abt, 1969) were designed to aid in the evaluation 
of educational materials* 



The research literature boast^ literally thousands of studies demling with 
I research on most areas of the cognitive, affective, ^and psychomotor domains. Of 
-particular relevance to the evaluation of instructional materials are the summaries 

by Bracht (1969) and Cronbach and Snow (1969) and the book edited by Wittrock and 

Wiley (1970) ... , . ' ■ 



The point that all the aforementioned literature attempts ta^make is that 
mere checkliists provide insufficient infomation about instructi<^hal materials. 
The.evaluatjion of instructianal materials requires a system that takes the user 

nd needs into consideration, eKamines.^process and product, and renders 
about the effectiveness of a given piece. or set of materials for a 
specified garget ^audience under a specified set of conditions. The work of 
Stufflebeairi, Scriven, Bracht Cronbach and Snow^ Solomon ■ (ai72) arid/ . ^ /"'T 
Webster and Mendro (1974, 1975) , -is used in this, paper to "develop ^a^striight 
forward operational model for the "evaiuation of instructiona]. materials, / 



condition a| 
a decision 



A MODEL FOR JNSTRUCTrONAL MATERIALS EVALUATION 



Pigur^ 1.0 presents a flowchart for ah integrated model for evaluating in- 
,structional materials. The events depicted in Figure 1.0 may take place within 
the period' of one day, ten years, or a lifetime, depending on the scope of the 
materials /to be evaluated^ , . 



Figure 1.0 



Con t ex t_E valuation and Needs Assess ment 

' ' ti ' ■ . ' ,t ' • 

A prerequisi^te jto improvement must be a knowledige of existing performanca 
levels,. Thus, befdr^ beginning the search for instructional materials ^ you must /; / 
establish a need for such materials . Stuffl^eam et ai. (1971) define content . ^ ^■ 
evaluation as the provision of baseline Information that delineates the environment ' 
of interest, describas desired and actual conditions pertaining to the envirortment. 
Identifies unmet needs and iinused opportunities, and diagnoses the problems that 
prevent needff^-from being^ met and opportunities from being used. Thus an adequate^ 
conteKt evaluatioh would point to problems that heed resolution ^ such as low reading 
-^math achieyementi, high student dropout, high teacher turnover, low student or 
i'tefstaff morale, and so on. An eKample of an operating context evaluation ^stem wa 
presented by Websjter and Sehuhm'acher (1973). 

■ . : ■ ■ \ ,.8' " 
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Once the context evaluation system has identified needs, those needsXmust be 
prioritized, and mafiagement must fqcus upon reducing the discrapancy between desired 
and eKisting conditions by establishing gokls for highest priority needs, it should 
be obvious^^Kat thp^ felt need to select new materials'^ properly begins with dissatis=- 
f action^^^lth those in use. In order for you to be legitimately dissatisfied, however, 
you must at least suspect that current materials are not meeting needs. The fact 
that there is a discrepancy between desired outcomes and needs suggests the ^^existence 
of objectives or_ goals. Thus a loop is f ormed^'-contejct evaluation^ needs assessment, 
goal setting. . Or is it the other way around? Denton (1976) presents a concise 
discussion of the complex interrelatedhess of these three mutualiy dependent concepts. 
Some methodology for needs assessment has been discussed by the Center for the Study 
of Evaluation {1973}, Feldmesser (1973), the Florida Educational Research and Develop- 
ment Council (1958), K^nd Kaufman (1972), Bolin (1973)^, Cook and Walbesser (1973), Uhl 
(1971) and Weatherman^ and Swenson (1974) .have presented models for goal setting* 

Input Evaluation 

____ /Input evaluation hel^s deteonine the best' resources to accomplish program 
goals. It logically follows the context evaluatioh, needs assessment, goal setting 
stage. The program planner can use four ;maj or sources of information in selecting 
instructional materials to meet specif id needs i (lj| Previous sunmiative product evai- 
uation informatio^; (2) basic research informationi (3) applied research . informatipm 
and (4) non^empirical^tnformation- ■ I 

. f(l) Eximmative produpt evai^uation information concerns the extent to which . 
specific project or program goals are aihieved. When product evaluation information^ ^'^ 
is available on a given program with goals similar to those identified in response 
to context evaluation^ information, that information helps decision makers determine , 
the probability that ^he program would reduce the iienti^fied discrepancy between 
desired and enisting conditions, , ; 

(2) Ba%ia research information pertains to fundamental, relations!, ips that " ] 
affect student, learning. . Before making a decision- to implement 4 given program, 
decision makers should know if that program is or is not consistent with the principles 
established by basic research in learning and developnsnt* ^ 

(3) Applied research information concerns the interaction between student 
characteriptics, teacrier-;' characteristics, arid instructional systems* Applied'' 
research differs from baaic research in that the information is more closely related 

to specific decisions in an applied educational setting. Decision makers need info±ma-fe 
tion about the types of students (e.g*, high anxiety versus low anxiety) who function 
best in given instructional systems as implemented by teachers with different types 
of characteristics or traits, ■ , ' 

(4) Nonemplrical information alsb^^enters into any^ materials selection decision* 
The influence of .materials distributors, costs, political feasibility of material 
adoption in given communities, capabilities and attitudes of staff meitoers> and exlstr^ 
ing facilities are among the many nonempirica.1 considerations that influence such 
decikions^ . \ • ^ - , . - , \ 




/ 



Many checklists or evaluation fornis have, been developed for aiding educators 
in the materials selection process . These f oirms are useful for organising in£orm« 
atidn about instructional materials /but in themselves do not provide for sufficient 
materials' evaluation. In many cases such forms are used to summarise information of 
the type previously reference^* A particuiarly organized approach was presented by 
Ar^mstrong (1973Jj_^qther-instrumen^^ were developed by Carpenter and Froke (1968), 
Caldwell (1968), Kovac (1976), Grobe (1976), Miller/(1969) , and the National 
Institute q.f Education (Product Rating Form, 1974). ^ In addition, the Educational 
Products Information EKchange has published a nun^er of guidelines t;or product 
selection (Komoski, 1967, 197S; Educational Products InfoMiation Exchange, 1971) . 

Determination of Resources ' 



Mter the collection of relevant input information feeding the preliminary 
program planning stage,. it must be r'-terminad whether sufficient resources are 
available to make the desired inatrL ionai changes. Quite often, adequate resources 
are not available and some compromise is neWssary, In many cases, the lack of 
resources is not limited to the realm of cost and political feasibility but rather 
stems from an insufficient base of research informatid^n. Thus, educators are often 
in the position of having sufficient Material Xresouraes but insufficient informal 
tion resources* ~ ■ . 



If sufficient material ^resources kre not mvftilable, the system may have^to 
exist for some time in a su^itm of , enlightened persistence. Periodic contekt eval- 
uation will continue to highlight the — - l^j..-^- .: v. 
desired and that which/ exists. 



^tent of discrepancy between that wjiich is 



If sufficient information resourcea are jiot available, programs often are 
implemented without sufficient support dkta and an information base must be built 
through a series of systematic evaluation and applied research studies. In some 
cases , ^ development centers are establitehek to cope with the piroilem ^f insufficient 
inforTnation resources. These centers are \^responsible for developing ,Uns true tional 
. systemsv to meet the needs ■Qutlinedxby qonteKt evaluation, Mate^ials^' and instructional 
systems are developed at the local level only if no potentiallyj useful coiranercial^ 
materials are available, since the development of instructional' systems is an 
extremely costly proposition. Figure 2LQ q^utlines a developmental' process that theo- 
retically could be fol%5wed in developing instructional sy^emsL An excellent 
introductory reference on instructional product development waa idited by Bak^r and 
Schutg. (1971). / \ 



/ 



Figure 2 ..6 



Program Planning 



/ 



s a^a available, the e^te^ed program planning phase is 
lemantfftd using information gleaned from input evaluatipft^ 



If sufficient resource 
entered. This phase is x p 

The program planning stage includes the developme# of detailed program^ objectives 
and of a management plan, o^m product of the eK^iikded program plannln^^rre-is^^^ 
detailea program evaluation design specifying t^He oriteria-by^whrch'^t^ instructional 
materials wJ.ll be judged, / This design, and ita^:in^lfe^xitation should b^(done by an 
individual wto3^«te:^l^ witii p^ogram maji%:^en^ is independent of them to 
allow. .Jor rtikim^ flexi^Mitrr Webs dutlihed an operational procedure 
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for assuring the independence of. prdj^ct evaluators. 

■ 1 ' = " ■ . 

Program Implementation 



Once the program implementation phase ^ is antere^ the evaluator provides 
continuous formativa evaluation raports on program implementatipn. These reports : 
fall 'primarily into two catego^'ies^ proces^s evaluation and interim product avalua- 
tion. Process evaluation has three major objectives : (1) The detection or. predic'^ 
tion df defects in procedural design or Its implementation , during program v implement-' 
ation stages; (2) the provision of information for programme^^^^ the 
maintenance of a record of the implementation procedure as it occurs (Stuff lebeam/ 
Bt Btlj., 1971). Thus* process evaluation information keeps program management informed 
of : theyextent to* which prqgraifi implementation conforms to specifications and/ from 
an eva^luation standpoint, guards against i-he evaluation of a fictitious event. 
Methods of accomplishing this have been 'discussed by Ashburn (1975) , Denton (1976)^ 
Mac>^ (1975) ^,>Niedermeya^ (1972) , and Rosenshine and Furst (1973). 

'/ Interim product evaluation provides periodic feedback to program management on 
thf attaLrynent of specific siibj-objectives during the implementation phase. Thus, 
process. and interim product evaiuation reports inform program "managament as td ' 
implementation and goal attainment levels while program adjustments are still fea-^ 
sibl^^ Formative evaluation methodology; has been discussed by Bloom, Hastings, and 
iijs : (1971)^ Rossi (1969), and Scriven (1967) , \ ; 



Produot Evaluation 



^et o 



relat 
ation 



Upon completion of a given cycle of jprbgrarn implementation^ a summative product 
evaluation report is prepared. . This report generally addresses *three areas of concerns 
S'(l) he extent ^to which prbgram objectives were. achieved r^lativev to- sofne 
slf criteria I i the extent to which 



specific 

. . system objectives were achieved relative 

to alternative instructional strategies; and (3) the cost-ef fectilveness of the program 
ive to alternative instructional strategies. It should be obvious that inform^" 
on these three areas ..must be iTi.|^rpreted in light of ■ process, >§nd interim product' 
evaldation iriformation. WiEhpufe informatioa^/ab program implementation*^ product 
^valujitiori information is of ^littie use. - . \ 



f experimental design 
i, resources have bean 



Figure 3.0 outlines the necessary steps in product evaluation. -The reader 
should bear in mind that product evaluation depe^ids on strong 
and commences at that point^that the'' need has been identified^. ^, 

allocated, and the program has been tentatively planned* Product Evaluation method- 
plo^ was the primary focus of many of the evaluation models discussed earlier. 
Useful articles on various aspects of prdduot evaluation include\ those by Alkin (1970), 
Baker and Schutz (1972)^ and Webster (1971) . In addition, some good-' texts on exper'- 
imental design and applied statistics include those by Campbell and Stanlay U966) , 
Edwards (1972) , Finn (1974) , Glasg and Stanley (1910) # arid Tatsuoka (-1971), ^ 



Figure 3,0 



P^p lied Re s e a r ch 



/ / 
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varying treatment configurations. Unfortunately, as a result, most product evaluatibn 
reports generally have focused on the search for single best treatments for all 
learners, i*e., main effects, in order to provide needed informatipn for educatidnal 
decision makers, applied research studies ^involving the systematic investigatipn of. 
aptitude-treatment and trait--trait interactions must be under tnken* *Such studies 
would be expected to provide important information on replicable relationships among 
student, teacher/ and program characteristics (Bracht, 1969i Cronbach and Snow, 1969i 
Solomon, 1972j Webster and Mendro^ 1974) • , r : 

The basic assumption of. aptitude- treatment interactipri ^rese^rch is that learners 
possess characteristics or traits that interact positively or negatively with specific 
treatments or program characteristic^'. Messick .(1970) outlined some cognitive-style 
dimensions that represent a -person's^ typical modes of perci^iving,^ remenODering , think- 
ing, and problem solving, and, as s^uch, would provide excellent' variables for aptitude- 
treatment interaction studies. In addition j many ^affective variables warrant in- 
vestigation when attempting to validate replicable teacher-student (trait-trait) 
interactions. The basic a'ssumptton of , trait-trait interaction research is that 
teachers possess characteristics or traits that, independent of program, interact 
positively or negatively with specific characteristics or traits possessed by learners. 
Such interac tions may involve variablei such as arithmetic reasoning, language usage, 
yocabulary^ abstract reasoning, mechanical ^reasoning, creativity , anxiety ^ af filia- 
tion ^ aggressiveness, compulsiveness, dogmatism, paranoia, and status variables such - 
as sex, age, or ethnicity* 

Nonempirical Information | / . 

Once. context/ input, process, ■ and product evaluation infonnation, as well as 
applied research data, are available, nonempirical information once rtore is l^rougfit : 
to bgar upon the decision-making process. It would be naive to expect educational v 
decisions to be made purely on the basis of empirical data. Once again, information 
such as the absolute progr am costs, capabilities of program staff members, political / 
feasibility of program implementation, and existing facilities and resources must be ' 
considered by dicision inakers* / 

Determination of Program Fate " . . / 

In determining the fate of a given program, four primary choices are available 
to decision makers i to continue, discontinue, expand, or revise the ^^ogram, ? First, 
they can choose to continue the program in its current setting. If this alternative 
is chosen, the summative product .evpluat^ and the applied research kata ' 

become the context evaluation information^^fbr^the next^^im^^ 

program implementation ^aommences . This alternative generallV occurs jf^rhen decisions 
are to/ be made on ..^he basis of , longitudinal studies, i*e*, where it is expeated^^^-- 
that results will not be in evidence after a relatively short implementation period, 

A second alternative is to discontinue the program* This is uaual'ly /dojie a£ter^ % 
produGt evaluation studies demonstrate the failure of the program to meels[it^ 
objectives or in those cases where the program is simply hot cost-^ef fectiiy^e. (Failure' 
to meet objectives is of ton a necessary, but not sufficient, condition for program 
discontinuation.) Once a program is discontiniied, the system returns to/ the context 
evaluation phase and once again applies the noeds assessment and orientation phases. 

■ " / / : ■ 
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■ - / ■ ' ' ' " - 

Thirds if the product evaluation and applied research information are favor- 
able and if it is practiGally and politically feasible, the program may be expanded 
,to serve additional students. Prior to eKpandlng the program , additional context - 
evaluation information must be examined to determine if similar needs exist in other 
settings. If such needs are demonstrated, then the program planning stage .^is 
:fntered to extend the program implementation. Other settings eligible for program 
' ekpansion may include entire schools or specific sutopopulations Ce*g* , highly 
motiyated students) as indicated by applied research data. If sucli needs 4o/not 
exist, the program is continued with the original target population or a: reduced 
target population based on the results of the applied research studies: Th^ extent 
of continued evaluation undar eitiier the expansion or Gontinuation alternative is 
determined by decision makerj with advice from evaluation personnel* i ■ 

A fourth alternative involves program revision. Much program revision should 
be accomplished on the basis of process and intbri-m product evaluation repcjrts* 
Often, however, summative product evaluation and applied research reports i^eveal 
weaknesses in portions of programs that would other^^ise appear to be functional, . 
In this instance, the summative product evaluation and applied research reports 
become the context evaluation information for the next program=^planning cy^le* - 

. ' ^ - ■ ^ ■ ■ ^ ^ - \ . ^' ^ * I " 

A PRACTICAL K<AMPLE ' ; 

This .evaluation model was used in the Dallas irideperident School District's ) 
Targeted Achievement in Reading Program (TARP) , Context evaluation data, compiled 
in 19.71, demonstrated a clear ijeed for reading intervention in a nun^er of the » 
District's schools. Among the data identifying such a need werei 

■ ■■■ 

1, Extremely low scores^ in reading and language on standardized 
^ . tdsts, usually at tHe chince level, j ... ^{ 

2t Extremely low scores -in reading and language on the District's' 
criterion^ref erenced tests, 

.1 , ■ ' / . ^ , ' ' ■ . 

3. High ptudent mobility, ^ . 

4. Low student attendance, ' 

5. High teacher turnover, 

6* LOW student motiyation, ^ ^ 

. 7, Parent apathy. ' ^■ 

Thus the need was established for programs that wouldi (1) Teach baslo reading 
and language skills to a highly mobile populationi (2) be sufficiently intarosting 
to increase student abtendance and motivationi (3) have a parents-training component 
to increase parent interest; and (4) have a teacher-development ""com^nent to in- 
crease teacher skills and motivation in dealing with disadvantaged students, thereby 
rdducimj teacher turnover* 
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A search was begun for progranis designed to enhance individualized insteuc-- 
: tioa that pbssassad the four characteris tics listed above. Program planners used' 
infonnatiQn that had worked in other areas [preVious product evaluatiun] / suggestions 
t^afc should work [basic research] > and ^ where avatl^lei the characteristics of j 
environments in which specific programs . had met their objectives [applied research], 
Coiranimity and staff opinions also were considered [non- research and evaluation infoi^ 
ma^ion] . Four programs were chosen^-three on tte bas^s of all four considerations 
Jprevious product evaluation^ basic resMrch dnfomation/ applied research information^ 
and non-research and evaluation information] , and one solely on the basis of opinion 
[non-research and evaluation information] ) ^wittiout reference to previous product 
evaluation information thkt suggested it would not work in the desired environment* 
Cost considerations were not foremost in the decision since the District was primarily 
concerned with finding material resources necessary to meet the demonstrated need^ 

^ Once tlie four programs were chosen^ managers were appointed ^ and operational 
program pl^nU^g commenced. Program manager^< with technical assistance from 
evaluation staff, developed specific objectives and management pl^s to accomplish 
those bbjectives* Evaluators^ respons^le'^to the Department ^f Research and 
'Evaluation, and program facilitators, responsible to the program managers, were 
assigned and began implementing their respective tasks*, The program facili%ati&^s 
implemented th^ar programs accordil^/^o the management plans, and the e4aluatoirs 
reported on howxc lose project impi'ernentation was to planned .„implemen.totion [process 
avaluation] and^n.how weld the respe^^ve programs were doing relative |to specific 
sub-objectives [irtterim product eva\luatx^n] . These reports were pj^duced ^at i . 
regular intervals and were used by proj#^^_jnanagem6nt to make in»caurse adjustments * 
in the projects. Many of the interim procesl^Md product evaluation\ reports were \^ 
In the fOCT of memoranda* . . ' \ 

, since it was predicted that several years 'wo^ld be required to implement the 
programs in the schools, the TAHP project involved a three-year longitudinal ' 
evaluation design. That ip, decisions about the fate of the programs would be made 
on the basis of three years of. evaluative data* - In-course adjustaients would be 
made on the basis of interim arid annual evaluation re^rts* . 



The product evaluation reports examined four basic questions about each of 
the four programs* These q^estiona werei 

" 1- Did each of the programs meet its objeotiv#s? / , . ' o 

- 2v Hpw did the programs eSrtipare to each otiher in terms of meetiilg 

District and national objectives in reading and ranguage? 

3, How did th6 programa compare'to each other in terms of 
increasing student attendance and motivation # decreasing 
teacher turnover , and increasing parental interest? .} 

4* How cost-effective were the programe? 



In addition, applied research Infermatlon on the types of teacher-student aaii^ina- 

tions that produced the beat results in each<6f the programs was extoined. 

Variables such as cognitive c^illty^ anKiety/# B^K, ethnio baakgroundi agei eKperienoe'f 



trainiingi 



attitude toward ins trtaqt ion, racial attitudes, creativity , social class. 



mobility, and eGonomiG level of both teachars. and students \mr^ investigated, with 
soma very enlightening. results , For instancey it was discovared that certain types 
of studand 



Alt] 



s do better in some pfog^ams when taught by specific types of ^teaaha^s , 



iough designed to run for three years ^ one of the four programs was 
.judged sol clearly inferior on the basis of the^ established criteria- that it was 
eliminatad after two years .\^as it turhad out/.the progijam/ilimina was ttte qna 
that was chosen originally ^ithout considM data^^ Thus two ; " 

years of tesources were wasted because of insuffiGient or inadequate input inf or- 
matioh. I ■ ■■ ^ Y ^ . \^ 

:&r 



Three prbgrams' remained! af&r three years of implamen|ati6n/ with each 



N 

52 'addi=N 
grams first v/ere 

he programs ware found similarly \suGcessful in meeting t^iir-vobjectives 
of the Dist^iGt. They produced similar patteirns of perf^Tnance oiim 



program about ten schools, 
tional echdols of similar co^ 
tried* 
and thoi 



The TARP programj than was to fiq^ expandad to 
text to the 30 schools where the'toograms fli 



staAdarcJizad tests arid similar outcomes on conGomitant vari^les/ (attendance ,\ 
motivation, teacher turnover, parental interest)* . Th^ quastions of interest w^rei 



(1) Which programs were to be eliminatadi (2) which were to be eKpaindedi and 
which, lif any^ were to Gontinue operating in the Gontext in which they ^had 
been implamanted* ' ; i 



(3\ / 
tginally 



/ since all the prdgrams achieved similar results, none was" to be eliminated on the 
basis bf; outcomes,. The next question was that of ^nput, or Gost, All tl^ree pro-^ // 
grams J cost abQut the sama to. operate, once they were established. However, one 
progrAm Gost about three times/ as much as the other two to Implamant* Thus, the 
cost-effectivenass question yielded additional information for decision-making. ' 
It wqfuld not be particularly psaful to eliminate the expensiva program, since once 
implemented it cost approxima^tely the same as the other two (and applied research 
information suggested that i^e more structured approach "was more effeGtive with 
high anxiety students )* Yet^' its additional implementation cost would not warrant ^ 
itr/ expansion. Therefore^ it was decided to keep that program in the schools : / 
where it had been originally .implemented, and, when po^siblei to assign high anxiety 
students to it. \ ' ^ . . ' 



I The other two programs achieved about the same results and cost approKitnately 
the same* Therefore, those two programs were e^^anded to the 52 additional schools^ 
with school faculties and eonmunities choosing the program they preferred,! Thus* 
school communities were permitted freedom of choiqe within a restrioted set of two 
alternatives, either of which had a high probability \of success. , . \ 



Once the programs passed out of the proje'ct evaluation ^hasa they were con- 
tinuously monitored through system-wide oontext evaluation* Curirentlyi if achievement 
results or .other varl^les in the participating schools display ^daerementQ^ the 
evaluation proaess is re-ontered, and replacement programs or strategiee are. 
initiated on a sampling basis* ' i ' 



SUMMARY 

^ ' This paper has a^ttemptad to smmarige the state, of the art in the evaluation 
/of iristructional materials; The position has bean that such evaluation should be 
implemtoted in a systematic manner and shouW include cohtakt, input, procesB/ and 
product evaluation with applied and basic research input,! The relevant evaluatioa 
literature has been SOTimarized, and an operational model (or evaluating instructional 
matarials tiiat could be implemented in a day or^a lifetime has been presented/ along 
with an example df it^v usefulness * For further information, the reader is encoiwraged 
to pursue readings from the following list of references* Two very good general 
references on evaluation are Anderson, Ball, and Murphy (1975) and Worthen ,and 
Sanders (1973)^ and some excellent evaluation iinits are operating in the Cincinnati # 
Dallas, and Philadelphia public school systems* 



Alkin* -U^ C, "Evaluating Net Cpst-Effectiveness of ihstructional Programs 
In ThB/EvMlumtlon of Instruationi Isbubb and Problmms, edited by C* 
Wittrock and 1. Wiley/ pp. 221-238. -New Yo^ki Holt, Rinahart and 
.Winston, 1970. \ . ' /', ' .. ' ' / ^ , 

Draws the distinction, between , cost^benef it ^^a^ cost-ef feGtiveness 
analysis and outlines GOrftponents of a mod^l for conducting cost^ 
ef fectivenass avaluationf in aducation* The. model considers student 
inputs f educational outputs, finanQial^-lnputs, sKternal systems, and 
' man ipulat able characteristics, ^Potential appiications of the model 
are noted* , / ^ ' ^ ^ j 

Alkin, M, C* , and Wooley, D* C, ModBl fat EduamtibMl Evml^atlon^ 
P^per presented at P1£DGE Conference, San Diinas, California^ .^Odtober 8-11, 
1969. 12pp,/ ED^036 898* V 

.The primary source for the Decision Oriented Classification Modal. 
>4neflnes evaluation as the process of ascertaining' deoisiqn areas oi 

conceE'n, selecting appropriate infomation) and cpllecting and analyzinc 

information in order to report summary data useful to decision^raakers 

in selecting among alterhattves'* 

Anderson, S* B*; Ball, Sy> and Mu^hy, T, Enayalopedla of EducatlanBl 
Evaluation: Cona&ptm and Techniques for Evaluating Education mnd Twining 
Programs^ San Francisqo^ Jossey--Bass, 1975* 

Provides an excellent overview of tte field of educational = e^raluatiori 
^ in terme that are generally oomprehenaible to non- specialists* ' 

Armstrong, J. R. , ed. a Sourcebook for the Evaluation of InBtruatlonal 
Materials and Medla> Arlington, Virginia i IMC/RMC Network ;Madlson t 
University of Wisconsin, Special Education instruotional Materials Center, 
August, ^1973* 520pp. ED 107 050.. ^ ^* 

Describes a generalized model for the evaluation of instructional 
materials and media; designed to be usable hy thm ihstructional 
Materials Centers/Regional Media Centers/Regional Resourca Centars 
Natworkp /Tl^e author discuss procedures for%aelecting 

instructional materials at various stages of the ^valuatipn process. 
AppendiKes include a b^liography and ^ eat' of forms for evaluating , . 
instructional materials and media* ^ . ' 

Ashburn, A* G* "Analysis of Aativity-Decision Flow in Prooese Evaluation 
Design*" EduaatiohalTmahnology, 15 (June, 1975), 36-39, 

Praaents a generalised activity-decision flow diagram deiineating tha 
types of proceaa evaluation information needs that ara'^likaly to arise 
in the course of project implementation. 

Baker, R. , and Bchutai R* E., ede* InstruQtlohai Product Drnvmlopmnt^ 
7 vole* Southwest Regional Laboratory for Eduqational Research and 
Development* New Yorkt Van Noatrand Ralnhold, 1971. 

An introductory text, includes Instructional Bequances dealing with 
stating educatipnal objectives, conetructing objeatlvea of cognitive 
behavior,' daveVtiping instruational epaaifioatlona, educational criterion 
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, . . .. ;. • = ^ , . . ■■ ,^ ■ 

measures, rules for the development of instructional products, preparing 
-r- .instructional products, and managing classroom contingencies.^ / 

Baker, L. ^ and SchutE, R. E. , eds. Instructional Produat Besearah.- 
8 vols . Southwest Regional Laboratory for Educational Research and / / 
Development , New York i yan' Nostrand. Reinhold, 1972, / . 

. An introductory text. Includes instructional sequences dealing wi'tli 
classifying and interpreting^ "research studies, selecting variables, for ^ 
Vrasearch, proposal writing, simplified research design, simplified 
analysis, the use of library computer progr^s for statistical analysis, 
and writing research reports. . ^ 

Bloom, B. S . ; Hastijigs , J, T.| and Madaus, F. Handbodk on Formative 
and Sunmativm, EvmluMtion of Student Learning. New YorkT McGraw-Hill, ' 

Provides an introductory overview^ of the use of evaluatibifi dh/ education, 
•nie final 11 dhaptars deal with the application of eWluation/ tio 11 
different subject areas on levels ranging frbm preschool education to 

Boiih^ ;J. ' **aix Crjfteria for Bettor Goals. " Improving Col lege and . 
UTvLv^rBlty .Teaching, 21 (August, 1973) , 245-247. , . ^ 

/ Posits siK criteria for developing adequate goals. Criteri^a include^ , 
; compatibility, attainability, intelligibility , accept^il^ty, 
; measurability, and accountability* 4 

^ • . — ■ - . ■ ^ ■ ^ , ■ ■ ■ \ 

Brac^t, G, H, The Relationship of Treatment TaBkBy PerBonalog^fiaal . 
Variablesr and Dependent Variables to Aptltude-Treainnent InteraGtlons / 
jDoctoifal dissertation. University of Colorado, Boulder, Colorado, 1969, 
Available from- University Microfilms, 300 N". ^eeb Road, Ann Arbor, 
Michigan 48105 (Order No, 70-05820; $15. 00 papercopy, $7.50 microfiche) . 
Provides an excellent review of the research on aptitudes-treatment 
interactions. ^ 

Caldwell, M. S. Jnpufc Evaluation and Mutational Flan^ 
The Ohio State University, Evaluation Center, Janua^, 1968. ZSpp* 
ED OiS 043. ^ ^1 

Suggests eight criteria that might be used as a screening system .for 
instructional materials. Criteria included relevance, legality,. 
/ congruence, legitimacy, compatibility, balance,, practicability, and 
cost-ef f ectivene 



Bill D. , and Stanley, J. C. ^^perimehtal and* QmBi^E^perlmental 
Designs for Research^ Chicago i Rand^McNallyi 1966, 

The olassla work on applied eKperimehtal design. Applies the 
principles of^ eKperimental design to field settings. * ^ 

Carpenter, C. R. , -and Proke, M. DeBarlption of a Practiaa^l Proaedurm for 
AsBeBBlng InBtruatlonal Film and TolmvlBion Programs, UniVerBity Parki 
The Pennsylvania State Universityi bepartment of Paytohology# August, 1968. 
■43pp. ED 037 102, , ' \ 

Provides a aystom of ch#ckllets for rating instructional' materlale. . 

Although designed for use with inst^uational film and television programs 



it would^need only slight jmpdifications to be^use^ with other 
^ instructional materi^s* ■ 



>'e^ter for i^the Study ofvivaluatiori/ Slmmentajcy School Evaluation Kiti 

I N&dds AsBmsBmmnt^ Bostons Aliynland Baconr 1973, 

Introdubtory level training^ materials dealing -w^ the goal selectiori 
process/ tfe^t seleqtion, Qata /CQlleGtion/ and the identification of 
. aritical naeS areas. | ; . " 



*CQok/ J^JU.f and l^albeBl^^ M. How to Mmet Accoiintmhillty with ^ 
3Bhmvioral Obj&Gtivms' and^Learning Himrarchims^ Colu'ntoia/ Maryland: 
Advahce Eduaational ^ress,\!973* 
; Discusses the use of beha^^doral" objectives and learning hierarchies 
— in evaluation. / i \ , . 



CranS/ .P, / and Abt^ C* "A Model for Currioultmi Evaluation." Edua&tionml 
' Technology ^ 9 J October, 196^) /^17-25. . ..\ ' 

The^ primary source for the Weightad Criteria Approach to curriculiMn 
evaluation* Based on GOst--ef feet i vanes s techniques^ applied to 
curriculum materials in order to analyze componmts, qualityV and 
cost of the materials being clpnsidered. "1-^^ 

Cronbach, J,, and Snow, E^,^ Individual DiffferencmB in Lmaming 
as a. Function of Instruatlonal Variables. Filial R&port^ Stanford 
California^ Stanford Universiti^/1 Schdor o£ Education, March/ 1969. ^221pp, 
ED 029 001* . - ■ ' I . ^ ' r 

: A careful review of relevant J.iteratur#. in thb -area of research on 

learning and individual differences. Provides ^methoddlogical ' 
auggestions^ for better aptituda-treateient in^ interaction studies. 

*Denton, W* T. Context Evaluation f\^NmedB Assessment,^ and Goal Setting i 
The InfinitB Loqp^ Paper presented at Evaluatioh" of Instructional Materials 
Workshop/ AECT Conventiori, toaheirap California/ Marph, 1976. \ 
Discusses the relatiynship and methodology of cdnteKt evaluation/ 
needs assessment/ and goal setting* ^ . , . ">.•'. . 



*Denton/ T. The Design and Implem&ntmtldn of Process and Interim Product 
Evaluation, Paper presented at Evaluation of Instruational Materials . / 
Workshop/ AECT Convention/ Anahei^/ California/ Mardh/ 1976. ' / 

Diicusses the role and inethodop,ogy of process and interim product / 
r evaluation. Contends tiiat formative evaluation of this type is the / 
/ moat important form of evaluation. Provides applied examples, / 

Educational Products InforTOatioAj EKchange Institute. "Looking Ahead: 
Selection Kits Will Blunt Buying Blunders. " Muca tiona J Product "Jn 
^B^ief*' Report Nurt^er 32, 4 (FebKijary/ 1971)/ 5-6. / 

Disaussea EPIl! Kits that include techniquea for avaluatlhg and 



sources of information for se 
equipment . 



looting inetructionial materials and 



EdwardS/ A. L. ^ Esiperimentml Dbb 

New Vorki Holt/ Rinehart^nd Winston/ 19^2. ' ) 
\ ^ introductory text on appll " - 



ign in Psyghologiaal ReBmrah» 4th ed. 
nston/ 19^2. ■ ) ^ , 

ed statistics and experimental design 



^Contact William J» Webster/ Depulty Associate Superintendent/ Research/ 
Evaluation/ and Information SyatSpma,, S. ij. Hay Building/ 3001 Hersohel 
Avenue/" ballai/ Texas 7S220. 
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|for behavioral scientists*, Cpncentrates on infarantial statistics 
ithrough univariate analysis bf variance/ with careful attention to 
. ^axperimental design. Particularly thorough coverage of analysis 
iof variance for an introductory text. • i 

•■ ■ ■ . ' ' ' ■ ^ ■ ■ ■ ■ ^ ■ 
Feldmesser, A. ; Eduoational 

New Jersey 1 Educational Testing Service, February ^ 1973. 68pp* ED 077 945, 



DesGription of a^ratipnale 



Goal Indlamtors fb^ tlew Jersey^ Prindaton^ 



for the needs assespment program carried 



out by the State Department i of New Jersey, 

Fihn^ J. D* A Gmneral Model for MultivMrl^tm Analysis. ^ New Yorki Holt, 
Rinehart and Winston, 1974. - \ ^ . _ 

An intarinediate level applied statistics text. on\ multivariate analysis. 
The book relies entirely on the statement /^d formulation of linear 
models to solve; multivariate problems. Closely aligned tq the 
MULTIVARIANCE Gomputer program, it uses san^le problems to illuatrkte 
various analysis techniquei^^ from multiple regression analysis tiirpugh 
; factor, analysis,, to multivariate eHtension of arialysis of variance, 
analysis of covariance, and regression analysis. ^ - 



Florida Educational Research land Development Coiincil, p; 
of the Edugational Needs of Florida. Gainesville i Univ 
1968- . V 

Outlines a plan for a statewide needs assessment in Florida 



n for Study 
sity of Florida, 



Glass, G. V. "Educational Product Evaluation I h Prototype \FQrpffat Applied, 

Eduaatipnml Researcher, 1 (1972) , -Iff 4, v . ^ 

The primary sourco for the Wadc^off and Comparative? Cost^^pproach 
to evaluation. Presents a model that could be used in . a^ar^ihing 
instructional materials foij, posstole adoption. 



r and Stanley, J, C* StatiBtlcal MBthodB in Edacmt^on and' 
Englewood Cliffs, New Jersey # Prentice-Hall, 1970. 



Glass, . G, V, 
Psyahology * 

h particularly thorough introductory text on applied univariate 
statistics and experimental design\ Discusses descrlpti^^ and\ 
inferential statistics through , analysis of variance,' 

*Groba^ R, Jnput Evaluatiorir Planned Developwflnt p^ 
StrategieB and Programs*^ Pap^^ presentea at Evaluation of Instructibnal 
Materials Workshop, AECT Convention, Anaheim^ Ca^lifornia, March, ,1976\^ 

Diseusaes ttie methodology and purpo^4 of input evaljuktion. Provides 

applied eKamples of the process, 

Hunter^ M, G* , and. Schooley, D- 1. ^ T^He BynBrgistla Evaluation Model* 1 V 
Paper prae anted "^t the Annual Meeting of the Amarioan Educational Research. 
Assoaiation, New Orleans, Louisiana, February 25-MarahIl| 1973. 10pp. 
ED Ot5 503, . * / / . _ 

The primary soUree for thertSynergistic Evaluation Model. Defines, 



evaluation as a process for 
deaision-making , 



athering and providing infpmation for 



♦Contact William J. Webitori .Deputy Aisociate iupe^intendent# Research, 
•Evaluation, and Information systema/ s. J. Hay Building, 3801 Herschel 
Avenue, Dal^laSi TexaB 75220. ^ ' ■ ' 
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^ufmanS R» Eiucmtidnkl System^ Planning^ Englewodd Cliffs^ New J'erseys 
^■--entice^Hall, 1972, ■ V V - / ^J^--' ^ ^ '/ 'I ' 



An introductory tekt oh educational planning J 



Komoski, K* '^Instructibnal Maeerialss Do They or Don't Theyi" Learning 
^ 3 (March, 197b) , 92-93. / =\ , ' / ^ ^ 

Describes a means for providing a systematic flow of irifoimation to 
, help teachers and schools evaluate and select those teaching/learning 

materials that *are mosf likely to work well with speciMc prbgrams . 
and students- ' / ' • / ^ 

Kompski^ K. DevBlQpm&nt of & System for mn Educatldnal Wraduats 
Information ExahangeJ Final ^R&port. New York^ Educational Products 
InfoCTnation Exchange Institute, Noven^er, 1967. 187pp, ; ED 020 566. 
Outlines anj eag^ly accesstole nationwide system^ for eKChan^ 
descriptiveL eValua&ve product information am^hg all sectors of the 
. educationalj commimity oh a cooperative |'cost-sfiaring)^basis . riie 

feasibility/ of such an arrangement educational t^ofessionals " ' 

is investigated as a selection base for curriculim use or desiqn 
improvement. . / ■ 

^ •■ ^ ,s ' , •■ ^ . : ;^ ^ . 

Kovac, R*/ Contact persbh for information on selectirig and evkluating non-" 
print materials (1976). Head of Selection, Catalog and Informatibn . Services ^ ^ 
Purdua Audio-visual Center f West Lafayette, Indiana ■47907. \ 

^Lamrock, A. L,; Smithy A. D,; and Warren^ P. W* Evaiumtion: Its Soopk and 
Syktmm for Evaluating PevWopinent. /bttaw^i New Start^ Corporation, March/. 1971* 
The primary source for the New vStart Evaluation Mbdi^l* This model focuses. , 
on, providing infomation to course developers so that they might make \ 
appropriate decision^ to accepti modify, or rejeat developed components; 
obtain information for dissemination activities ;/ and. obtain: information 
to aid in product installation.,. / 

Lindvall/ C, M*, and CoK^ The IPI Evaluatim Program, AERA Monograph Series 

on Curriculum Evaluation. Chicago i Rand McNaaly, 1970* 

h primary source fo^ the Individually Pre^ribed Instruction (IPI) . ^ 
Evaluation Model. This modei focuses oh providing reli^le information 
for decision'-making during the development stages of innovati'^a progrsuns-.. 



Magy^ D* J* "The Role of Process Evaluation in Program Development and 
Implementation^ " Eduaatianml Tmalmologyf 15 (April, 1975) > 42-47 * 
\ An applied discussion q£ the role of prbeess evaluation in program 
development and jjnplementation. ^ 

Messick/ S* "The Criterion Problem in the Evaluation of Instructions 

Aaaesaing Possible, Not Just Intendedi butcomea," In The Bvalumtlon of. 

InBtruatloni I&muBB and Problem&i edited by M. C. wittrock and D. 

Wllay^ pp. 183-204. New yorki . Holt, Rinehart^^nd Winsteon* 1970. 
Discusses cognitive styles and affective reactions as two major 
elaeses of oriterion variabloi that should be conslderod in the 
evaluation of instruction. 



*Contagt William J. Websteri Deputy Asayoiate Suparlntondent, Researchi 
Evaluation, and Information Systems ^ a. J. iray Building^ 3001 mrmgh^l 
AvenuQi DallaBi /PyKaB 7S220. ' ' ' 



t^illerr E. I, Critmria for Assfsslng Instructional Units. Lexingtoni 
University of Kentuekyrl Departs of Social and Philosophical Studies in 
Education; October/ 1969. Project funded by the Fund for Media Researoh, 
General Learning Corporation^ , and the KaroK Corporation. ; 

Suggesta a plan for assassing" instructional strategies usirig 11 * 
specific briteria. For each oi the criteria a nratber of quastiona' 
are ^-isted ^for ,;gonsidetation in the overall analysis* / ; ; 
' ' ' - ' T ' ^ ^ ' '-^ . " ^ ' ^* ■ ■ 

Morgan^ R* L. An Appromah to Evaluations A Model for Evaluahlng the ' 
North Carolina'' EKemplary Progrmm, Raleigh: l*0rth Carolina State . . 
University, Center for dccupatiqnal Education* TJuty, JL970. Paper\ - 
presented at the National Institute on EKemplary Projects in Vocational 
Education^ Squaw Valley, California, Jul^^ 1970. ^26ppi ED 042 910,;, 
. T?nB-prima^ source for the Apex Evaluation ModelT It gives jio J ^ 
eKplicit definition of evaluation, but iinplies that evaluatioflh^ . 
exists to serve the decision-^making process. . ' \ ^ ' 

■ ■ ^ '" . ■ ' . '■ ' ^ ' ,^ ' ^ ' • * ^ ' ■ 

National Institute of Education* NIB Prbduat Rating Form* Washington, J^Vc.r 
National Institute bf Education, 1974. 14pp* /So be availabla?^, through ERIC* 
The fprm used .by the National Institute of Efluda^ipnr to facilitate\ 
review of the 'quality of educational products* This form was \ 
developed to indicate ditnensions of apparent , quality even when \ • 
highly credible evidence is not available to confim the ratings*: \ x 
The form is quite generaliz^le^ -X^ ^ , * • , 

Niedermeyer, C* Obsetyation Prppedures f6r. Clas^room/^Tryout^of'^ , 
Inmtruatiohal Material^^a^Q^^PrgcedureB^ I^s Alamttos, GalifQrhiar ^ ' ' ' 
Southwest Regional I^aboratory, for EducatiSnal Research and Development, 
June, 1972* ISpp, ^ED 106? 887* ^ f . ^ 7^ 

Suggasts, certain procedures for reporting elasaroom observation of . 
a program les^uii* Thecc procedtires are illustrated in the . y 

observation of two, laboratory programs j , "Composition Skills" and 
"Drama and Public Speaking*" ■ . , . % 



Ott, J* M* ; Fletcher, S.? and Turner ^ D* G* "Taxonomy of Administrative 
Information Needs i topAid to Educational Planning and Evaluation*" * 
Eduaation^al technology, 8 1 5 (1966), 29-31* 

The primary source for ^e Ott Evaliiation Model* Presents a 
classification system/of^'decieion eituatlons that must be served ■ 
' by Evaluation. ' j * 

Peper# J» B* An Qnto],Qgiaal MbdGl of Evalumtioni Dy^nmmio MpdmX fo^ 
Aiding Organisational Developments Paper presented at the Annual Meeting / 
of. the American Educational Research Association, New Orleans, ^ulsiana, 
February 25-March 1^ 1973* 'SSpp* ED 078 039* ' . \ 

Tha.priipary source fo^c^^ithe Ontological Evaluation Model* This model ^ 
focuses on the organisation as a whole # rather than on individual s 
programs within the organization*. 

ProvuB, M. M. DimarepMnay Evmlmtioni^ , For BduaatSoml Program Improv&mmnt 

and ABBBBBmeht, BerkQley, Californias McCutqhan, 1971* 

The prlinary source for the pisotfapangy Evaluation' Model. Defines / ' 
ovaluation as ^th©^ process of defining. program standards, determining 
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whetliar a discrepancy exists between some aspect of program perforrnatice 
and standards^ and using discrepancy infOCTiation to change perforniance 
or to change program standards. ^ 

Rosenshine, B. , and Fursti N» "The Use of DireGt Observation to Study 
Teaching*" In Second Handbook of RBseMrch on Teaching, edited by 
iravera, pp, 122-183. " Chicago s Rand McNally, ^ 1973. . 

Pr'Qvides a revievr* of the literature pn the use of direct observation 
schema to study teoching. An excellent source on classroom observation 
procedures and models. ^ - , , 

Rossis P. HV Practice, Method and llieory^ in Evaluating Social Action 
Programs."- In On Fighting Poverty ; PBTspectives from Experience , edited 
by j, Suhdquist, pp/ 219-234. New Yotfks Basic Books^ 1969, 
Discusses the role of formative evaluation in evaluating social 
programs , . " 

Scriven, M. "The Methodology of Evaluation*" In Perspactives of 
Curriaulum Eimluatlon, AERA Monograph #1^. edited by- R, Tyler, R, Gagne, 
and Scriven. Chicago: Rand McNally, 1967, Out of prints . 

Focuses on^ curricular evaluatidhV ^' Brscuases, among other topics, 

the roles of formative and summative evaluation in prbviding ^ 

information for curriculum development and validation , 

Salomon, s.> "Heuristic Models /^or the Generation of Aptitude^Treatment 
Interaction Hypotheses." Review of Educational Researah, 42 (1972) , 
327=343* ^ ^ 

Presents three heuristic models for the design of aptitude-^treatment 

ihteraction studies* . ' ' 

Stake, R, E>^. "The Countenance of Educational Evaluation," Teachers 
College Redbrd,. 68 (1967) ^ 523-540. ' 

The primary source for the Countenance of Educational Ev^aluation, 
Suggests that evaluation ought to be concerned witii three classes 
of conditions I antecedents, . transactions^ and outcomes, 

^Stuf f lebeam, D, L. , ed* Eduaational Eyaliimtiop and Deal sion ^Making. 

Itaska, Illinois: E* Peacock Publishers^ 1971* 

_The primary source for the discussion of the CIPP (cont^Kt, input, 
process^ and product) model for educational evaluation* Defines 
evaluation, as the Requisition of infomation, through formal means, 
such as criteria, measuLrement, and statistics # to prbvi'de ^rational 
bases for arriving at judgmenta' in decision-making* . . ' 

Tatsuoka, M* MultivarlBte Aiialysis^J v^yechnigua^^ for Educational mnd 
pBychologlaal ResBarch*' New Yorki John- Wiley and Sons, 1971, 
An ;jpplied text on multi'^ariate analysis X Deals cleariy and' ^ 
concisely with such topics as multiple r4gresslon, multivariate 
analysis. of variance and covariance, rtfinqipal components analysis, 
discriminant aftalysis, and canonical cprr elation* 
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Tripodi, T- 1 Pellin^ P.; and Epstein, I. Social Prpgram Evmluationi 
Cuidelines for HeMlth^ Education, and Welfare Mdministrmtors . Itaska, 
Illinois: F. E. Peacock Publishers, 1971, 

The primary source' for the discussion of the Differential Evaluation 
Modal. Dafines evaluation as the systematic accmnulation of facts 
for providing information about the achievement of program requisites 
and goals relative to efforts, effectiveness, and efficiency within 
any stage of program developmeht. Such facts are incorporated into 
some designated value system for making decisions about social programs, 

Uhl/ N. P. Encouraging ConvergencB of Opinion^ through the Use of DQlphi 
Techmique, in the Process of idBntifying an Institution' s Goals. Princeton, 
New Jerseys Educational Testing Service, February, 1971. 126pp. 
ED 049 713- < ' ' ^ 

Discusses 'the' Delphi Technique as a method for gaining convergenca ' 

of opinion*! ' ; 

Weatherman, R. , and Swenson, K. 1 "Delphi Technique^" In Futurism in 
Educations Methodologies, edited by S* P^ Hencley and J* Yates. 
Berkeley, Mlifornia:/ McCKitchari, 1974* 

Prasents^tha methodology of the Delphi Technique, which is designed 



to encourage convergence of opinion* 

Webste;r, W*. J. Thm Organization knd Functions of ResBaich mnd Evaluation 
in a Largs Urban Sahool Districts \Paper presented at the Annual Meeting 
of the American Educational Research Association, Washington, D*C*, March 
30^-^ril 3, 1975, 72pp, ED 106 345* 

Outlines the purpose, organisation, and functions of research and 
' evaluation in a large urban school district* Major issues discussed 
include methods of ensuringthe credibility %f evaluation information, 
the establishment and maintenance of meanirigfiul raportjUig cycles, - 
^the recruitment and employment of qualified personnel, the release 
- and dissemination of evaluation and research da%a/'ahd the relatlc 
ship between research, evaluation, plahhing, and developmant* 

Webster, W* J* "Technical Auditing Procedures*" In EQuvatlonal Produat ' r 
Audit Handb^- ' pp. 38^103* Arlington,. Virginiai Institute for the ' ^ 

Deyelopman .ijaucational Auditing, 1971* , ^ ' \ . - . 

PKovid s f.n overviaw of procedures and issues in evaluation design nnd 
a taKon^my of statistical techniques for use in specific problem 
situatirns* ' ^ ' 



Webster, W* J.., and Mendro, R* "The Investigation of Aptitude-Treatmerit 

Interactionr is an ^Integral Part of Program Evaluation," Journal of 
E^perimsntaj. 'ducation, 43 . (Fall, , 1974) J 86-&1* 

' Presents model for integrating aptijtuda-trGatinent and trait-trait ■ 
, interaction studies into pfogram evaluation* . , 

*Webster, W. J*, and jMendro, 3* L* "A Pragmatic Model for a Comprehensive 
Piablic School' Research and Evaluation S _ 
AmBearch, 69 {December, 1975), 156-1S9. . ' 

' Present^ an aK^lled i^del for planning, reaearuh,i^and evaluation*' ' ^ 
Incorpi^rates context, input, processi-, and produ^' ©valuation, as 
Well as basic and applied research, « 
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Webster/ W. J,, and Schuhinacher, C. C. "A Unified Strategy for SyStain-Wide 
Research and Evaluation." Eduaational Technology, 13 (May, 1973) , 68-'72 J > 
Discusses an operational context evaluation system designed to provide 
dGcision--makers with important baseline information about student, 
staff, and conmunity characteristics^ 

Wittrock, M. C. , and Wiley, D, Thm Evaluation of Instruction: Issues 

and Prablams, New Yorkr Holt, Rinehart and Winston, 1970. 

Rexjorts on thm outcomes of a symposium designed to bring new approaches 
to the probleitts and issues of instructional evaluation. Topics 
include: evaluation theory, instructional variables, contextual 
variables, instructional criteria^ and evaluation methodology. 

V/orthen, B» R. , and Sanders, j. Educational Evaluations Theory and 

PraaticB. Belmont, Californiai Wadsworth, 1973. 

A book of readings that includes . many of the classic works in the 

field of educatiorial evaluation p 



The material in this publication was prepared pursuit 
to a con tract with the'^National Institu^m of Education^ 
United states DBpartmBht of Healthy Education ^ and 
WBlfare. Contractors undertaking such projBcts under 
governmBnt sponsorship are encouraged to express freely 
their judgement in profess ional and technical matters. 
Prior to publication f tliis manuscript was submitted to 
the executive director' qf the Educational Products 
Information Exchange Institute^ ^or 'critical review and 
determination of professional competence. This publica- 
tion has met such standard^: Points of view or ppinionst 
howeverf do not necessarily represent the official view 
oryopinians of either the reviewer or the National 
Institute of Education^ Contract No^ NIE-C-^74-0027 » 
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